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This program, bzip2, the associated library 1ibbzip2, and all documentation, are copyright © 1996-2007 Julian Seward. All rights reserved. 


Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 


¢ Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 


¢ The origin of this software must not be misrepresented; you must not claim that you wrote the original software. If you use this software in a 
product, an acknowledgment in the product documentation would be appreciated but is not required. 


¢ Altered source versions must be plainly marked as such, and must not be misrepresented as being the original software. 
¢ The name of the author may not be used to endorse or promote products derived from this software without specific prior written permission. 


THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT 
NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DIS- 
CLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, 
OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, 
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT 
OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 


PATENTS: To the best of my knowledge, bzip2 and libbzip2 do not use any patented algorithms. However, I do not have the resources to 
carry out a patent search. Therefore I cannot give any guarantee of the above statement. 
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1. Introduction 


bzip2 compresses files using the Burrows-Wheeler block-sorting text compression algorithm, and Huffman coding. 
Compression is generally considerably better than that achieved by more conventional LZ77/LZ78-based compressors, 


and approaches the performance of the PPM family of statistical compressors. 


bzip2 is built on top of libbzip2, a flexible library for handling compressed data in the bzip2 format. This 
manual describes both how to use the program and how to work with the library interface. Most of the manual is 
devoted to this library, not the program, which is good news if your interest is only in the program. 


¢ How to use bzip2 [2] describes how to use bzip2; this is the only part you need to read if you just want to know 


how to operate the program. 
¢ Programming with libbzip2 [8] describes the programming interfaces in detail, and 


¢ Miscellanea [31] records some miscellaneous notes which I thought ought to be recorded somewhere. 





2. How to use bzip2 
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This chapter contains a copy of the bzip2 man page, and nothing else. 


2.1. NAME 


* bzip2, bunzip2 - a block-sorting file compressor, v1.0.4 
* bzcat - decompresses files to stdout 


* bzip2recover - recovers data from damaged bzip?2 files 


2.2. SYNOPSIS 


* bzip2 [ -cdfkqstvzVL123456789 | [ filenames... ] 
* bunzip2[-fkvsVL ] [ filenames... ] 
* bzcat [-s][ filenames... ] 


* bzip2recover filename 
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2.3. DESCRIPTION 


bzip2 compresses files using the Burrows-Wheeler block sorting text compression algorithm, and Huffman coding. 
Compression is generally considerably better than that achieved by more conventional LZ77/LZ78-based compressors, 
and approaches the performance of the PPM family of statistical compressors. 


The command-line options are deliberately very similar to those of GNU gzip, but they are not identical. 


bzip2 expects a list of file names to accompany the command-line flags. Each file is replaced by a compressed 
version of itself, with the name original_name.bz2. Each compressed file has the same modification date, 
permissions, and, when possible, ownership as the corresponding original, so that these properties can be correctly 
restored at decompression time. File name handling is naive in the sense that there is no mechanism for preserving 
original file names, permissions, ownerships or dates in filesystems which lack these concepts, or have serious file 
name length restrictions, such as MS-DOS. 


bzip2 and bunzip2 will by default not overwrite existing files. If you want this to happen, specify the —f flag. 


If no file names are specified, bzip2 compresses from standard input to standard output. In this case, bzip2 will 
decline to write compressed output to a terminal, as this would be entirely incomprehensible and therefore pointless. 


bunzip2 (orbzip2 —d) decompresses all specified files. Files which were not created by bzip2 will be detected 
and ignored, and a warning issued. bzip2 attempts to guess the filename for the decompressed file from that of the 
compressed file as follows: 

* filename.bz2 becomes filename 


¢ filename.bz becomes filename 


¢ filename.tbz2 becomes filename.tar 





e filename.tbz becomes filename.tar 
* anyothername becomes anyothername. out 


If the file does not end in one of the recognised endings, .bz2, .bz, .tbz2 or .tbz, bzip2 complains that it 
cannot guess the name of the original file, and uses the original name with . out appended. 


As with compression, supplying no filenames causes decompression from standard input to standard output. 


bunzip2 will correctly decompress a file which is the concatenation of two or more compressed files. The result is 
the concatenation of the corresponding uncompressed files. Integrity testing (-t) of concatenated compressed files is 
also supported. 


You can also compress or decompress files to the standard output by giving the —c flag. Multiple files may be 
compressed and decompressed like this. The resulting outputs are fed sequentially to stdout. Compression of 
multiple files in this manner generates a stream containing multiple compressed file representations. Such a stream 
can be decompressed correctly only by bzip2 version 0.9.0 or later. Earlier versions of bzip2 will stop after 
decompressing the first file in the stream. 


bzcat (orbzip2 -dc) decompresses all specified files to the standard output. 


bzip2 will read arguments from the environment variables BZIP2 and BZIP, in that order, and will process them 
before any arguments read from the command line. This gives a convenient way to supply default arguments. 


Compression is always performed, even if the compressed file is slightly larger than the original. Files of less than 
about one hundred bytes tend to get larger, since the compression mechanism has a constant overhead in the region of 
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50 bytes. Random data (including the output of most file compressors) is coded at about 8.05 bits per byte, giving an 
expansion of around 0.5%. 


As a self-check for your protection, bzip2 uses 32-bit CRCs to make sure that the decompressed version of a file is 
identical to the original. This guards against corruption of the compressed data, and against undetected bugs in bzip2 
(hopefully very unlikely). The chances of data corruption going undetected is microscopic, about one chance in four 
billion for each file processed. Be aware, though, that the check occurs upon decompression, so it can only tell you 
that something is wrong. It can’t help you recover the original uncompressed data. You can use bzip2recover to 
try to recover data from damaged files. 


Return values: 0 for a normal exit, | for environmental problems (file not found, invalid flags, I/O errors, etc.), 2 to 
indicate a corrupt compressed file, 3 for an internal consistency error (eg, bug) which caused bzip2 to panic. 


2.4. OPTIONS 


-c —-stdout 
Compress or decompress to standard output. 





d decompress 
Force decompression. bzip2, bunzip2 and bzcat are really the same program, and the decision about 
what actions to take is done on the basis of which name is used. This flag overrides that mechanism, and 
forces bzip2 to decompress. 


-zZ —-compress 
The complement to —d: forces compression, regardless of the invokation name. 


= --kest 
Check integrity of the specified file(s), but don’t decompress them. This really performs a trial decompres- 
sion and throws away the result. 


-E =“Pores 
Force overwrite of output files. Normally, bzip2 will not overwrite existing output files. Also forces 
bzip2 to break hard links to files, which it otherwise wouldn’t do. 


bzip2 normally declines to decompress files which don’t have the correct magic header bytes. If forced 
(—f£), however, it will pass such files through unmodified. This is how GNU gzip behaves. 


-k --keep 
Keep (don’t delete) input files during compression or decompression. 


-s --small 
Reduce memory usage, for compression, decompression and testing. Files are decompressed and tested 
using a modified algorithm which only requires 2.5 bytes per block byte. This means any file can be 
decompressed in 2300k of memory, albeit at about half the normal speed. 


During compression, —s selects a block size of 200k, which limits memory use to around the same figure, at 
the expense of your compression ratio. In short, if your machine is low on memory (8 megabytes or less), 
use —s for everything. See MEMORY MANAGEMENT [5] below. 


-q --quiet 
Suppress non-essential warning messages. Messages pertaining to I/O errors and other critical events will not 
be suppressed. 
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-v --verbose 
Verbose mode -- show the compression ratio for each file processed. Further —v’s increase the verbosity 
level, spewing out lots of information which is primarily of interest for diagnostic purposes. 





L licens Vv version 
Display the software version, license terms and conditions. 


—1 (or --fast) to —9 (or -best) 
Set the block size to 100 k, 200 k ... 900 k when compressing. Has no effect when decompressing. See 
MEMORY MANAGEMENT [5] below. The --fast and —-best aliases are primarily for GNU gzip 
compatibility. In particular, --fast doesn’t make things significantly faster. And ——best merely selects 
the default behaviour. 


Treats all subsequent arguments as file names, even if they start with a dash. This is so you can handle files 
with names beginning with a dash, for example: bzip2 myfilename. 











repetitive-fast, repetitive-best 
These flags are redundant in versions 0.9.5 and above. They provided some coarse control over the behaviour 
of the sorting algorithm in earlier versions, which was sometimes useful. 0.9.5 and above have an improved 
algorithm which renders these flags irrelevant. 


2.5. MEMORY MANAGEMENT 


bzip2 compresses large files in blocks. The block size affects both the compression ratio achieved, and the amount 
of memory needed for compression and decompression. The flags —1 through —9 specify the block size to be 100,000 
bytes through 900,000 bytes (the default) respectively. At decompression time, the block size used for compression 
is read from the header of the compressed file, and bunzip2 then allocates itself just enough memory to decompress 
the file. Since block sizes are stored in compressed files, it follows that the flags —1 to —9 are irrelevant to and so 
ignored during decompression. 


Compression and decompression requirements, in bytes, can be estimated as: 
Compression: 400k + ( 8 x block size ) 


Decompression: 100k + ( 4 x block size ), or 
HOO GZ bilo eke suse) 


Larger block sizes give rapidly diminishing marginal returns. Most of the compression comes from the first two or 
three hundred k of block size, a fact worth bearing in mind when using bzip2 on small machines. It is also important 
to appreciate that the decompression memory requirement is set at compression time by the choice of block size. 


For files compressed with the default 900k block size, bunzip2 will require about 3700 kbytes to decompress. 
To support decompression of any file on a 4 megabyte machine, bunzip2 has an option to decompress using 
approximately half this amount of memory, about 2300 kbytes. Decompression speed is also halved, so you should 
use this option only where necessary. The relevant flag is —s. 


In general, try and use the largest block size memory constraints allow, since that maximises the compression achieved. 
Compression and decompression speed are virtually unaffected by block size. 


Another significant point applies to files which fit in a single block -- that means most files you’d encounter using a 
large block size. The amount of real memory touched is proportional to the size of the file, since the file is smaller 
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than a block. For example, compressing a file 20,000 bytes long with the flag —9 will cause the compressor to 
allocate around 7600k of memory, but only touch 400k + 20000 * 8 = 560 kbytes of it. Similarly, the decompressor 
will allocate 3700k but only touch 100k + 20000 * 4 = 180 kbytes. 


Here is a table which summarises the maximum memory usage for different block sizes. Also recorded is the total 
compressed size for 14 files of the Calgary Text Compression Corpus totalling 3,141,622 bytes. This column gives 
some feel for how compression varies with block size. These figures tend to understate the advantage of larger block 
sizes for larger files, since the Corpus is dominated by smaller files. 

Compress Decompress Decompress Corpus 








Flag usage usage -s usage Size 
=i 1200k 500k 350k 914704 
=2 2000k 900k 600k S77 10s 
=3 2800k 1300k 850k 860338 
-4 3600k 1700k 1100k 846899 
=) 4400k 2100k 1350k 845160 
—6 5200k 2500k 1600k 838626 
=7 6100k 2900k 1850k 834096 
-8 6800k 3300k 2100k 828642 
=) 7600k 3700k 2350k 828642 


2.6. RECOVERING DATA FROM DAMAGED 
FILES 


bzip2 compresses files in blocks, usually 900kbytes long. Each block is handled independently. If a media or 
transmission error causes a multi-block .bz2 file to become damaged, it may be possible to recover data from the 
undamaged blocks in the file. 


The compressed representation of each block is delimited by a 48-bit pattern, which makes it possible to find the 
block boundaries with reasonable certainty. Each block also carries its own 32-bit CRC, so damaged blocks can be 
distinguished from undamaged ones. 


bzip2recover is a simple program whose purpose is to search for blocks in .bz2 files, and write each block out 
into its own .bz2 file. You can then use bzip2 —t to test the integrity of the resulting files, and decompress those 
which are undamaged. 


bzip2recover takes a single argument, the name of the damaged file, and writes a number of files 
recO001file.bz2, rec0002file.bz2, etc, containing the extracted blocks. The output filenames are 
designed so that the use of wildcards in subsequent processing -- for example, bzip2 -dc recxfile.bz2 > 
recovered_data -- lists the files in the correct order. 


bzip2recover should be of most use dealing with large .bz2 files, as these will contain many blocks. It is clearly 
futile to use it on damaged single-block files, since a damaged block cannot be recovered. If you wish to minimise 
any potential data loss through media or transmission errors, you might consider compressing with a smaller block 
size. 


2.7. PERFORMANCE NOTES 


The sorting phase of compression gathers together similar strings in the file. Because of this, files containing very 
long runs of repeated symbols, like "aabaabaabaab ..." (repeated several hundred times) may compress more slowly 
than normal. Versions 0.9.5 and above fare much better than previous versions in this respect. The ratio between 
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worst-case and average-case compression time is in the region of 10:1. For previous versions, this figure was more 
like 100:1. You can use the —-vvvv option to monitor progress in great detail, if you want. 


Decompression speed is unaffected by these phenomena. 


bzip2 usually allocates several megabytes of memory to operate in, and then charges all over it in a fairly random 
fashion. This means that performance, both for compressing and decompressing, is largely determined by the speed 
at which your machine can service cache misses. Because of this, small changes to the code to reduce the miss rate 
have been observed to give disproportionately large performance improvements. I imagine bzip2 will perform best 
on machines with very large caches. 


2.8. CAVEATS 


I/O error messages are not as helpful as they could be. bzip2 tries hard to detect I/O errors and exit cleanly, but the 
details of what the problem is sometimes seem rather misleading. 


This manual page pertains to version 1.0.5 of bzip2. Compressed data created by this version is entirely forwards 
and backwards compatible with the previous public releases, versions 0.1p12, 0.9.0 and 0.9.5, 1.0.0, 1.0.1, 1.0.2 and 
1.0.3, but with the following exception: 0.9.0 and above can correctly decompress multiple concatenated compressed 
files. 0.1p12 cannot do this; it will stop after decompressing just the first file in the stream. 


bzip2recover versions prior to 1.0.2 used 32-bit integers to represent bit positions in compressed files, so it could 
not handle compressed files more than 512 megabytes long. Versions 1.0.2 and above use 64-bit ints on some 
platforms which support them (GNU supported targets, and Windows). To establish whether or not bzip2recover 
was built with such a limitation, run it without arguments. In any event you can build yourself an unlimited version if 
you can recompile it with MaybeUInt 64 set to be an unsigned 64-bit integer. 


2.9. AUTHOR 


Julian Seward, jseward@bzip.org 


The ideas embodied in bzip2 are due to (at least) the following people: Michael Burrows and David Wheeler (for 
the block sorting transformation), David Wheeler (again, for the Huffman coder), Peter Fenwick (for the structured 
coding model in the original bzip, and many refinements), and Alistair Moffat, Radford Neal and Ian Witten (for 
the arithmetic coder in the original bzip). Iam much indebted for their help, support and advice. See the manual 
in the source distribution for pointers to sources of documentation. Christian von Roques encouraged me to look 
for faster sorting algorithms, so as to speed up compression. Bela Lubkin encouraged me to improve the worst-case 
compression performance. Donna Robinson XMLised the documentation. Many people sent patches, helped with 
portability problems, lent machines, gave advice and were generally helpful. 
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This chapter describes the programming interface to Libbzip2. 


For general background information, particularly about memory use and performance aspects, you’d be well advised 
to read How to use bzip2 [2] as well. 


3.1. Top-level structure 


libbzip2 is a flexible library for compressing and decompressing data in the bzip2 data format. Although 
packaged as a single entity, it helps to regard the library as three separate parts: the low level interface, and the high 
level interface, and some utility functions. 


The structure of 1ibbzip2’s interfaces is similar to that of Jean-loup Gailly’s and Mark Adler’s excellent zlib 
library. 


All externally visible symbols have names beginning BZ2_. This is new in version 1.0. The intention is to minimise 
pollution of the namespaces of library clients. 
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To use any part of the library, you need to #include <bz1lib.h> into your sources. 


3.1.1. Low-level summary 


This interface provides services for compressing and decompressing data in memory. There’s no provision for dealing 
with files, streams or any other I/O mechanisms, just straight memory-to-memory work. In fact, this part of the library 
can be compiled without inclusion of st dio.h, which may be helpful for embedded applications. 


The low-level part of the library has no global variables and is therefore thread-safe. 


Six routines make up the low level interface: BZ2_bzCompressInit, BZ2_bzCompress, and 
BZ2_bzCompressEnd for compression, and a_ corresponding trio BZ2_bzDecompressInit, 
BZ2_bzDecompress and BZ2_bzDecompressEnd for decompression. The *Init functions allocate 
memory for compression/decompression and do other initialisations, whilst the *End functions close down 
operations and release memory. 











The real work is done by BZ2_bzCompress and BZ2_bzDecompress. These compress and decompress data 
from a user-supplied input buffer to a user-supplied output buffer. These buffers can be any size; arbitrary quantities of 
data are handled by making repeated calls to these functions. This is a flexible mechanism allowing a consumer-pull 
style of activity, or producer-push, or a mixture of both. 


3.1.2. High-level summary 


This interface provides some handy wrappers around the low-level interface to facilitate reading and writing bzip2 
format files (.bz2 files). The routines provide hooks to facilitate reading files in which the bzip2 data stream 
is embedded within some larger-scale file structure, or where there are multiple bzip2 data streams concatenated 
end-to-end. 


For reading files, BZ2_bzReadOpen, BZ2_bzRead, BZ2_bzReadClose and BZ2_bzReadGet Unused are 
supplied. For writing files, BZ22_bzWriteOpen, BZ2_bzWrite and BZ2_bzWriteFinish are available. 


As with the low-level library, no global variables are used so the library is per se thread-safe. | However, if I/O 
errors occur whilst reading or writing the underlying compressed files, you may have to consult errno to determine 
the cause of the error. In that case, you’d need a C library which correctly supports errno in a multithreaded 
environment. 


To make the library a little simpler and more portable, BZ2_bzReadOpen and BZ2_bzWriteOpen require you to 
pass them file handles (F ILE*s) which have previously been opened for reading or writing respectively. That avoids 
portability problems associated with file operations and file attributes, whilst not being much of an imposition on the 
programmer. 





3.1.3. Utility functions summary 


For very simple needs, BZ2_bzBuffToBuffCompress and BZ2_bzBuffToBuffDecompress are provided. 
These compress data in memory from one buffer to another buffer in a single function call. You should assess 
whether these functions fulfill your memory-to-memory compression/decompression requirements before investing 
effort in understanding the more general but more complex low-level interface. 


Yoshioka Tsuneo (tsuneo@rr.iij4u.or. jp) has contributed some functions to give better zlib compati- 
bility. These functions are BZ2_bzopen, BZ2_bzread, BZ2_bzwrite, BZ2_bzflush, BZ2_bzclose, 
BZ2_bzerror and BZ2_bzlibVersion. You may find these functions more convenient for simple file reading 
and writing, than those in the high-level interface. These functions are not (yet) officially part of the library, and are 
minimally documented here. If they break, you get to keep all the pieces. I hope to document them properly when 
time permits. 
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Yoshioka also contributed modifications to allow the library to be built as a Windows DLL. 


3.2. Error handling 


The library is designed to recover cleanly in all situations, including the worst-case situation of decompressing random 
data. I’m not 100% sure that it can always do this, so you might want to add a signal handler to catch segmentation 
violations during decompression if you are feeling especially paranoid. I would be interested in hearing more about 
the robustness of the library to corrupted compressed data. 


Version 1.0.3 more robust in this respect than any previous version. Investigations with Valgrind (a tool for detecting 
problems with memory management) indicate that, at least for the few files I tested, all single-bit errors in the 
decompressed data are caught properly, with no segmentation faults, no uses of uninitialised data, no out of range 
reads or writes, and no infinite looping in the decompressor. So it’s certainly pretty robust, although I wouldn’t claim 
it to be totally bombproof. 


The file bz1lib.h contains all definitions needed to use the library. In particular, you should definitely not include 
bzlib_private.h. 


In bz1lib.h, the various return values are defined. The following list is not intended as an exhaustive description of 
the circumstances in which a given value may be returned -- those descriptions are given later. Rather, it is intended to 
convey the rough meaning of each return value. The first five actions are normal and not intended to denote an error 
situation. 


BZ_OK 
The requested action was completed successfully. 


BZ_RUN_OK, BZ _FLUSH_OK, BZ_FINISH_OK 
In BZ2_bzCompress, the requested flush/finish/nothing-special action was completed successfully. 





BZ_STREAM_END 
Compression of data was completed, or the logical stream end was detected during decompression. 











The following return values indicate an error of some kind. 


BZ_CONFIG_ERROR 
Indicates that the library has been improperly compiled on your platform -- a major configuration error. 
Specifically, it means that sizeof (char), sizeof (short) and sizeof (int) are not 1, 2 and 
4 respectively, as they should be. Note that the library should still work properly on 64-bit platforms 
which follow the LP64 programming model -- that is, where sizeof (long) and sizeof (void*) are 
8. Under LP64, sizeof (int) is still 4, so Libbzip2, which doesn’t use the Long type, is OK. 








BZ_SEQUENCE_ERROR 
When using the library, it is important to call the functions in the correct sequence and with data structures 
(buffers etc) in the correct states. 1ibbzip2 checks as much as it can to ensure this is happening, and returns 
BZ_SEQUENCE_ERROR if not. Code which complies precisely with the function semantics, as detailed 
below, should never receive this value; such an event denotes buggy code which you should investigate. 
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BZ_PARAM_ERROR 
Returned when a parameter to a function call is out of range or otherwise manifestly incorrect. AS 
with BZ_SEQUENCE_ERROR, this denotes a bug in the client code. The distinction between 
BZ_PARAM_ ERROR and BZ_SEQUENCE_ERROR is a bit hazy, but still worth making. 
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BZ_MEM _ 











BZ_DATA __ 


BZ_DATA 


ERROR 


Returned when a request to allocate memory failed. Note that the quantity of memory needed to decompress 
a stream cannot be determined until the stream’s header has been read. So BZ2_bzDecompress and 
BZ2_bzRead may return BZ_MEM_ERROR even though some of the compressed data has been read. 
The same is not true for compression; once BZ2_bzCompressInit or BZ2_bzWriteOpen have 
successfully completed, BZ_MEM_ERROR cannot occur. 





ERROR 


























Returned when a data integrity error is detected during decompression. Most importantly, this means when 
stored and computed CRCs for the data do not match. This value is also returned upon detection of any other 
anomaly in the compressed data. 








BZ_I0_ 


BZ_UN 


BZ_OUTBUFF_FULL 


ERROR_MAGIC 





As a special case of BZ_DATA_ERROR, it is sometimes useful to know when the compressed stream does 
not start with the correct magic bytes (’B’ 'Z’ ‘h’). 





ERROR 


Returned by BZ2_bzRead and BZ2_bzWrite when there is an error reading or writing in the compressed 
file, and by BZ2_bzReadOpen and BZ2_bzWriteOpen for attempts to use a file for which the error 
indicator (viz, ferror (f))is set. On receipt of BZ_IO_ERROR, the caller should consult errno and/or 
perror to acquire operating-system specific information about the problem. 











EXPECT 








EOF 





ED_ 


Returned by 


BZ2_bzRead when the compressed file finishes before the logical end of stream is detected. 





Returned by BZ2_bzBuffToBuffCompress and BZ2_bzBuffToBuffDecompress to indicate that 
the output data will not fit into the output buffer provided. 


3.3. Low-level interface 


3.3.1. BZ2_bzCompressInit 
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typedef struct { 
Cha nexccaeon, 
unsigned int avail_in; 
unsigned int total_in_1032; 
unsigned int total_in_hi32; 


char *«next_out; 
unsigned int avail_out; 

unsigned int total_out_1032; 
unsigned int total_out_hi32; 











void «state; 


WACUCL 2 (loeiiL ikove!)) (Awenaicl 3, abide » alinne)) - 
Viola (bozmree) (ond s+, Viord: * )r 
void *opaque; 

| OZ Siciesennp 


int BZ2_bzCompressInit ( bz_stream *strm, 
int blockSizel00k, 
int verbosity, 
int workFactor ); 


Prepares for compression. The bz_stream structure holds all data pertaining to the compression activity. A 
bz_stream structure should be allocated and initialised prior to the call. The fields of bz_st ream comprise the 
entirety of the user-visible data. state is a pointer to the private data structures required for compression. 


Custom memory allocators are supported, via fields bzalloc, bzfree, and opaque. The value opaque is 
passed to as the first argument to all calls to bzalloc and bzfree, but is otherwise ignored by the library. The 
callbzalloc ( opaque, n, m ) is expected to return a pointer p ton * m bytes of memory, and bzfree 
( opaque, p ) should free that memory. 


If you don’t want to use a custom memory allocator, set bzalloc, bzfree and opaque to NULL, and the library 
will then use the standard malloc/ free routines. 


Before calling BZ2_bzCompressInit, fields bzalloc, bzfree and opaque should be filled appropriately, 
as just described. Upon return, the internal state will have been allocated and initialised, and total_in_1032, 
total_in_hi32, total_out_1032 and total_out_hi32 will have been set to zero. These four fields are 
used by the library to inform the caller of the total amount of data passed into and out of the library, respectively. 
You should not try to change them. As of version 1.0, 64-bit counts are maintained, even on 32-bit platforms, 
using the _hi32 fields to store the upper 32 bits of the count. So, for example, the total amount of data in is 
(total_in_hi32 << 32) + total_in_1032. 


Parameter blockSizel100k specifies the block size to be used for compression. It should be a value between | 
and 9 inclusive, and the actual block size used is 100000 x this figure. 9 gives the best compression but takes most 
memory. 


Parameter verbosity should be set to a number between 0 and 4 inclusive. 0 is silent, and greater numbers give 
increasingly verbose monitoring/debugging output. If the library has been compiled with -DBZ_NO_STDIO, no 
such output will appear for any verbosity setting. 


Parameter workFactor controls how the compression phase behaves when presented with worst case, highly 
repetitive, input data. If compression runs into difficulties caused by repetitive data, the library switches from 


12 


Programming with libbzip2 





the standard sorting algorithm to a fallback algorithm. The fallback is slower than the standard algorithm by perhaps 
a factor of three, but always behaves reasonably, no matter how bad the input. 


Lower values of workFactor reduce the amount of effort the standard algorithm will expend before resorting to the 
fallback. You should set this parameter carefully; too low, and many inputs will be handled by the fallback algorithm 
and so compress rather slowly, too high, and your average-to-worst case compression times can become very large. 
The default value of 30 gives reasonable behaviour over a wide range of circumstances. 


Allowable values range from 0 to 250 inclusive. 0 is a special case, equivalent to using the default value of 30. 
Note that the compressed output generated is the same regardless of whether or not the fallback algorithm is used. 


Be aware also that this parameter may disappear entirely in future versions of the library. In principle it should be 
possible to devise a good way to automatically choose which algorithm to use. Such a mechanism would render the 
parameter obsolete. 


Possible return values: 
BZ CONFIG _ ERROR 


if the library has been mis-—compiled 
BZ_PARAM_ ERROR 

if strm is NULL 

or blockSize < 1 or blockSize > 9 

or verbosity < 0 or verbosity > 4 

or workFactor < 0 or workFactor > 250 
BZ_MEM ERROR 

if not enough memory is available 
BZ_OK 

otherwise 




















Allowable next actions: 
BZ2_bzCompress 


if BZ_OK is returned 
no specific action needed in case of error 


3.3.2. BZ2_bzCompress 


int BZ2_bzCompress ( bz_stream *strm, int action ); 


Provides more input and/or output buffer space for the library. The caller maintains input and output buffers, and calls 
BZ2_bzCompress to transfer data between them. 


Before each call to BZ2_bzCompress, next_in should point at the data to be compressed, and avail_in should 
indicate how many bytes the library may read. BZ2_bzCompress updates next_in, avail_inandtotal_in 
to reflect the number of bytes it has read. 


Similarly, next__out should point to a buffer in which the compressed data is to be placed, with avail_out indi- 
cating how much output space is available. BZ2_lbzCompress updates next_out, avail_out andtotal_out 
to reflect the number of bytes output. 


You may provide and remove as little or as much data as you like on each call of BZ2_bzCompress. In the limit, 
it is acceptable to supply and remove data one byte at a time, although this would be terribly inefficient. You should 
always ensure that at least one byte of output space is available at each call. 
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A second purpose of BZ2_bzCompress is to request a change of mode of the compressed stream. 


Conceptually, a compressed stream can be in one of four states: IDLE, RUNNING, FLUSHING and FINISHING. Be- 
fore initialisation (BZ2_bzCompressInit) and after termination (BZ2_bzCompressEnd), a stream is regarded 
as IDLE. 





Upon initialisation (BZ2_bzCompressInit), the stream is placed in the RUNNING state. Subsequent calls 
to BZ2_bzCompress should pass BZ_RUN as the requested action; other actions are illegal and will result in 
BZ_SEQUENCE_ERROR. 


At some point, the calling program will have provided all the input data it wants to. It will then want to finish up -- 
in effect, asking the library to process any data it might have buffered internally. In this state, BZ22_bzCompress 
will no longer attempt to read data from next_in, but it will want to write data to next__out. Because the output 
buffer supplied by the user can be arbitrarily small, the finishing-up operation cannot necessarily be done with a single 
call of BZ2_bzCompress. 


Instead, the calling program passes BZ_F INISH as an action to BZ2_bzCompress. This changes the stream’s 
state to FINISHING. Any remaining input (ie, next_in[O .. avail_in-—1]) is compressed and transferred 
to the output buffer. To do this, BZ2_bzCompress must be called repeatedly until all the output has been 
consumed. At that point, BZ2_bzCompress returns BZ_STREAM_END, and the stream’s state is set back to 
IDLE. BZ2_bzCompressEnd should then be called. 

















Just to make sure the calling program does not cheat, the library makes a note of avail_in at the time of the 
first call to BZ2_bzCompress which has BZ_F INISH as an action (ie, at the time the program has announced its 
intention to not supply any more input). By comparing this value with that of avail_in over subsequent calls to 
BZ2_bzCompress, the library can detect any attempts to slip in more data to compress. Any calls for which this is 
detected will return BZ_SEQUENCE_ERROR. This indicates a programming mistake which should be corrected. 




















Instead of asking to finish, the calling program may ask BZ2_bzCompress to take all the remaining input, compress 
it and terminate the current (Burrows-Wheeler) compression block. This could be useful for error control purposes. 
The mechanism is analogous to that for finishing: call BZ2_bzCompress with an action of BZ_F LUSH, remove 
output data, and persist with the BZ_FLUSH action until the value BZ_RUN is returned. As with finishing, 
BZ2_bzCompress detects any attempt to provide more input data once the flush has begun. 


Once the flush is complete, the stream returns to the normal RUNNING state. 


This all sounds pretty complex, but isn’t really. Here’s a table which shows which actions are allowable in each 
state, what action will be taken, what the next state is, and what the non-error return values are. Note that you can’t 
explicitly ask what state the stream is in, but nor do you need to -- it can be inferred from the values returned by 
BZ2_bzCompress. 
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IDLE/any 
Tllegal. IDLE state only exists after BZ2_bzCompressEnd or 
before BZ2_bzCompressInit. 
Return value = BZ_SEQUENCE_ERROR 
































RUNNING/BZ_RUN 

Compress from next_in to next_out as much as possible. 
Next state = RUNNING 

Return value = BZ_RUN_OK 








RUNNING/BZ_FLUSH 
Remember current value of next_in. Compress from next_in 

to next_out as much as possible, but do not accept any more input. 
Next state = FLUSHING 

Reburne vices sb Amr nUS HEOK 





RUNNING/BZ_FINISH 
Remember current value of next_in. Compress from next_in 

to next_out as much as possible, but do not accept any more input. 
Next state = FINISHING 

Return value = BZ _FINISH_OK 

















FLUSHING/BZ_FLUSH 
Compress from next_in to next_out as much as possible, 
but do not accept any more input. 
If all the existing input has been used up and all compressed 
output has been removed 

Next state = RUNNING; Return value = BZ _RUN_OK 

ence 

Next state = FLUSHING; Return value = BZ_FLUSH_OK 


FLUSHING/other 
Illegal. 
Return value = BZ_SEQUENCE_ERROR 


























FINISHING/BZ_FINISH 
Compress from next_in to next_out as much as possible, 
but to not accept any more input. 
If all the existing input has been used up and all compressed 
output has been removed 
Next state IDLE; Return value = BZ STREAM END 
enkisie 
Next state 

















FINISHING; Return value = BZ_FINISH_OK 


FINISHING/other 
Illegal. 
Return value = BZ_SEQUENCE_ERROR 


























That still looks complicated? Well, fair enough. The usual sequence of calls for compressing a load of data is: 


1. Get started with BZ2_bzCompressInit. 
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2. Shovel data in and shlurp out its compressed form using zero or more calls of BZ2_bzCompress with action = 
BZ_RUN. 


3. Finish up. Repeatedly call BZ2_bzCompress with action = BZ_F INISH, copying out the compressed output, 
until BZ_ STREAM END is returned. 














4. Close up and go home. Call BZ2_bzCompressEnd. 


If the data you want to compress fits into your input buffer all at once, you can skip the calls of BZ2_bzCompress 
(_..., BZ_RUN ) andjustdothe BZ2_bzCompress ( ..., BZ_FINISH ) calls. 


All required memory is allocated by BZ2_bzCompressInit. The compression library can accept any data at all 
(obviously). So you shouldn’t get any error return values from the BZ2_bzCompress calls. If you do, they will be 
BZ_SEQUENCE_ERROR, and indicate a bug in your programming. 














Trivial other possible return values: 
BZ_PARAM_ ERROR 
alse Hevea abs INI, fone Sexe Sey als: INIUMIE 








3.3.3. BZ2_bzCompressEnd 


int BZ2_bzCompressEnd ( bz_stream *«strm ); 





Releases all memory associated with a compression stream. 


Possible return values: 
BZ PARAM FRROR if strm is NULL or strm->s is NULL 


BAnOK otherwise 





3.3.4. BZ2_bzDecompressInit 


int BZ2_bzDecompressInit ( bz_stream *strm, int verbosity, int small ); 


Prepares for decompression. As with BZ2_bzCompressInit, a bz_stream record should be allocated and 
initialised before the call. Fields bzalloc, bzfree and opaque should be set if a custom memory allocator is 
required, or made NULL for the normal malloc / free routines. Upon return, the internal state will have been 
initialised, and total_in and total_out will be zero. 


For the meaning of parameter verbosity, see BZ2_bzCompressInit. 


If small is nonzero, the library will use an alternative decompression algorithm which uses less memory but at the 
cost of decompressing more slowly (roughly speaking, half the speed, but the maximum memory requirement drops 
to around 2300k). See How to use bzip2 [2] for more information on memory management. 


Note that the amount of memory needed to decompress a stream cannot be determined until the stream’s header has 
been read, so even if BZ2_bzDecompressInit succeeds, a subsequent BZ2_bzDecompress could fail with 
BZ_MEM_ ERROR. 











Possible return values: 
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BZ_ CONFIG _ERROR 

if the library has been mis-compiled 
BZ_PARAM ERROR 

if ( small != 0 && small !=1 ) 

or (verbosity <; 0 || verbosity > 4) 
BZ_MEM ERROR 

if insufficient memory is available 




















Allowable next actions: 
BZ2_bzDecompress 


if BZ_OK was returned 
no specific action required in case of error 


3.3.5. BZ2_ bzDecompress 


int BZ2_bzDecompress ( bz_stream «strm ); 


Provides more input and/out output buffer space for the library. The caller maintains input and output buffers, and 
uses BZ2_bzDecompress to transfer data between them. 





Before each call to BZ2_bzDecompress, next_in should point at the compressed data, and avail_in 
should indicate how many bytes the library may read. BZ2_bzDecompress updates next_in, avail_in and 
total_in to reflect the number of bytes it has read. 


Similarly, next __out should point to a buffer in which the uncompressed output is to be placed, with avail_out 
indicating how much output space is available. BZ2_bzCompress updates next_out, avail_out and 
total_out to reflect the number of bytes output. 


You may provide and remove as little or as much data as you like on each call of BZ22_bzDecompress. In the 
limit, it is acceptable to supply and remove data one byte at a time, although this would be terribly inefficient. You 
should always ensure that at least one byte of output space is available at each call. 


Use of BZ2_bzDecompress is simpler than BZ2_bzCompress. 


You should provide input and remove output as described above, and repeatedly call BZ2_bzDecompress until 
BZ_STREAM_END is returned. Appearance of BZ_STREAM_END denotes that BZ2_bzDecompress has detected 
the logical end of the compressed stream. BZ2_bzDecompress will not produce BZ_STREAM_END until all output 
data has been placed into the output buffer, so once BZ_STREAM_END appears, you are guaranteed to have available 
all the decompressed output, and BZ2_bzDecompressEnd can safely be called. 


















































If case of an error return value, you should call BZ2_bzDecompressEnd to clean up and release memory. 


Possible return values: 
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BZ_PARAM_ ERROR 


if strm 


is NULL or strm->s is NULL 


Ge Siciin—SenyeLil ibe < il 





BZ_DATA_ERROR 


if a data integrity error is detected in the compressed stream 








BZ_DATA 


E 


RROR_MAGIC 











if the compressed stream doesn’t begin with the right magic bytes 
BZ_MEM ERROR 
if there wasn’t enough memory available 


























BZ_STREA 


| END 











if the ] 





logical end of the data stream was detected and all 


output in has been consumed, eg s-->avail_out > 0 


BZ_OK 
otherwi 


se 


Allowable next actions: 
BZ2_bzDecompress 


if BZ_OK was returned 
BZ2_bzDecompressEnd 


otherwi 


3.3.6. BZ 





SS 


2_bzDecompressEnd 





int BZ2_bzDecompressEnd ( bz_stream *strm ); 


Releases all memory associated with a decompression stream. 


Possible return values: 
BZ PARAM ERROR 


ciksfeuesiterrarm 
BZ_OK 
otherwi 





is NULL or strm->s is NULL 


se 


Allowable next actions: 


None. 


3.4. High-level interface 


This interface provides functions for reading and writing bzip2 format files. First, some general points. 


e All of the functions take an int first argument, bzerror. After each call, bzerror should be consulted 
first to determine the outcome of the call. If bzerror is BZ_OK, the call completed successfully, and only 
then should the return value of the function (if any) be consulted. If bzerror is BZ_IO_ERROR, there was an 
error reading/writing the underlying compressed file, and you should then consult errno / perror to determine 


the cause of 





the difficulty. ozerror may also be set to various other values; precise details are given on a 


per-function basis below. 
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¢ If bzerror indicates an error (ie, anything except BZ_OK and BZ_STREAM_END), you should immediately call 
BZ2_bzReadClose (or BZ2_bzWriteClose, depending on whether you are attempting to read or to write) 
to free up all resources associated with the stream. Once an error has been indicated, behaviour of all calls except 
BZ2_bzReadClose (BZ2_bzWriteClose) is undefined. The implication is that (1) bzerror should be 
checked after each call, and (2) if bzerror indicates an error, BZ2_bzReadClose (BZ2_bzWriteClose) 
should then be called to clean up. 





¢ The FILE* arguments passed to BZ2_bzReadOpen / BZ2_bzWriteOpen should be set to binary mode. 
Most Unix systems will do this by default, but other platforms, including Windows and Mac, will not. If you omit 
this, you may encounter problems when moving code to new platforms. 





Memory allocation requests are handled by malloc / free. At present there is no facility for user-defined 
memory allocators in the file I/O functions (could easily be added, though). 


3.4.1. BZ2_bzReadOpen 


typedef void BZFILE; 





BZFILE *BZ2_bzReadOpen( int *bzerror, FILE +f, 
int verbosity, int small, 
void xunused, int nUnused ); 








Prepare to read compressed data from file handle £. £ should refer to a file which has been opened for reading, and 
for which the error indicator (ferror (f) )is not set. If small is 1, the library will try to decompress using less 
memory, at the expense of speed. 


For reasons explained below, BZ2_bzRead will decompress the nUnused bytes starting at unused, before starting 
to read from the file £. At most BZ_MAX_UNUSED bytes may be supplied like this. If this facility is not required, 
you should pass NULL and 0 for unused and nUnused respectively. 





For the meaning of parameters small and verbosity, see BZ2_bzDecompress Init. 


The amount of memory needed to decompress a file cannot be determined until the file’s header has been read. 
So it is possible that BZ2_bzReadOpen returns BZ_OK but a subsequent call of BZ2_bzRead will return 
BZ_MEM ERROR 











Possible assignments to bzerror: 
BZ_CONFIG_ERROR 
if the library has been mis-compiled 
BZ_ PARAM ERROR 
if f is NULL 
or small is neither 0 nor 1 
or ( unused == NULL && nUnused != 0 ) 
or ( unused != NULL && ! (0 <= nUnused <= BZ MAX UNUSED) ) 
BZ_ITO_ERROR 
Ie Ee o1 GE) ase nonzero 
BZ MEM ERROR 
if insufficient memory is available 
BZ_OK 
otherwise. 



































Possible return values: 
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Pointer to an abstract BZFIL!I 





IES 


ER lO ZeiatsOre eS) SB AmOK 


NUL 


L 


otherwise 


Allowable next actions: 
BZ2_bzRead 


ase lowienciaoue aL) BWA OK 
BZ2_bzClose 
otherwise 


3.4.2. BZ2_ bzRead 


alinic, IVs lowalRieyere! (( alisne ssloASuciclone, IBVAIILININ Blo, Waoskcl lowhe, Shite iets) )) 4 





Reads up to len (uncompressed) bytes from the compressed file b into the buffer buf. If the read was successful, 
bzerror is set to BZ_OK and the number of bytes read is returned. If the logical end-of-stream was detected, 
bzerror will be set to BZ_STREAM_END, and the number of bytes read is returned. All other bzerror values 
denote an error. 


BZ2_bz 


is possibl 


Read 














will supply len bytes, unless the logical stream end is detected or an error occurs. Because of this, it 


e to detect the stream end by observing when the number of bytes returned is less than the number requested. 


Nevertheless, this is regarded as inadvisable; you should instead check bzerror after every call and watch out for 





BZ_STR 





KAM 





END. 





Internally, BZ2_lbzRead copies data from the compressed file in chunks of size BZ_MAX_UNUSED bytes be- 
fore decompressing it. If the file contains more bytes than strictly needed to reach the logical end-of-stream, 
BZ2_bzRead will almost certainly read some of the trailing data before signalling BZ_SEQUENCE_END. To col- 
lect the read but unused data once BZ_SEQUENCE_END has appeared, call BZ2_bzReadGet Unused immediately 
before BZ2_bzReadClose. 






































Possible assignments to bzerror: 
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BZ_PARAM_ ERROR 





if b is NULL or buf is NULL or len < 0 
BZ_SEQUENCE_ERROR 

if b was opened with BZ2_bzWriteOpen 
BZ_TO_ERROR 

if there is an error reading from the compressed file 
BZ_UNEXPECTED_EOF 

if the compressed fil nded befor 


the logical end-of-stream was detected 
BZ_DATA_ERROR 















































E 


if a data integrity error was detected in the compressed stream 
BZ_DATA_ERROR_MAGIC 
if the stream does not begin with the requisite header bytes 
(ie, is not a bzip2 data file). This is really 
a special case of BZ _DATA_ERROR. 
BZ_MEM ERROR 








E 





























if insufficient memory was available 
BZ_ STREAM _END 














if the logical end of stream was detected. 
BZ_OK 


otherwise. 


Possible return values: 
number of bytes read 


ik bozerror as BAZLOK or BZ STREAM END 
undefined 


otherwise 











Allowable next actions: 

Collect data from bum, then BAZ bzRead ior BAZ bzRead@llose 
if bzerror is BZ OK 
collect data from buf, then BZ2_bzReadClose or BZ2_bzReadGetUnused 

if bzerror is BZ _SEQUENCE_END 
BZ2_bzReadClose 
otherwise 
































3.4.3. BZ2 bzReadGetUnused 
void BZ2_bzReadGetUnused( intx bzerror, BZFILE xb, 
void** unused, int* nUnused ); 





Returns data which was read from the compressed file but was not needed to get to the logical end-of-stream. 
xunused is set to the address of the data, and *nUnused to the number of bytes. 


*nUnused will be set to a 
value between 0 and BZ_ MAX _UNUSED inclusive. 








This function may only be called once BZ2_bzRead_ has 


signalled BZ _STREAM_END but before 
BZ2_bzReadClose. 








Possible assignments to bzerror: 
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BZ_ PARAM ERROR 
if b is NULL 
or unused is NULL or nUnused is NULL 
BZ_SEQUENCE_ERROR 
if BZ_STREAM_END has not been signalled 
or if b was opened with BZ2_bzWriteOpen 
BOK 


otherwise 



































Allowable next actions: 
BZ2_bzReadClose 


3.4.4. BZ2 bzReadClose 


void BZ2_bzReadClose ( int *bzerror, BZFILE xb ); 





Releases all memory pertaining to the compressed file b. BZ2_bzReadClose does not call fclose on the 


underlying file handle, so you should do that yourself if appropriate. BZ2_lbzReadClose should be called to clean 
up after all error situations. 


Possible assignments to bzerror: 
BZ_SEQUENCE_ERROR 
if b was opened with BZ2_bzOpenWrite 
BZ_OK 
otherwise 




















Allowable next actions: 
none 


3.4.5. BZ2_bzWriteOpen 


BZFILE *BZ2_bzWriteOpen( int *bzerror, FILE xf, 
int blockSizel00k, int verbosity, 
int workFactor ); 








Prepare to write compressed data to file handle f. £ should refer to a file which has been opened for writing, and for 
which the error indicator (ferror (£) )is not set. 


For the meaning of parameters blockSizel00k, verbosity and workFactor, see BZ2_bzCompressInit. 





All required memory is allocated at this stage, so if the call completes successfully, BZ_MEM ERROR cannot be 
signalled by a subsequent call to BZ2_bzWrite. 











Possible assignments to bzerror: 


22 


Programming with libbzip2 





BZ_CONFIG_ERROR 
if the library has been mis-compiled 
BZ PARAM ERROR 
Shae Ge SLs} IN(U/ENE 
Opole ck Sa eiOOke i ores lrochkSasziewkO Oks 
BZ_ITO_ERROR 
if ferror(f) is nonzero 
BZ_MEM ERROR 
if insufficient memory is available 
BZ_OK 
otherwise 























Possible return values: 
Pointer to an abstract BZFILE 


aie loasyieisione ale) BWA (OV 
NULL 
otherwise 





Allowable next actions: 
BZ2_bzWrite 


if bzerror is BZ_OK 

(you could go directly to BZ2_bzWriteClose, but this would be pretty, 
pointless) 
BZ2_bzWriteClose 

otherwise 


3.4.6. BZ2 bzWrite 


void BZ2_bzWrite ( int *bzerror, BZFILE *b, void «buf, int len ); 








Absorbs len bytes from the buffer buf, eventually to be compressed and written to the file. 


Possible assignments to bzerror: 
BZ_ PARAM ERROR 

if b is NULL or buf is NULL or len < 0 
BZ_SEQUENCE_ERROR 

if b was opened with BZ2_bzReadOpen 
BZ_TO_ERROR 

if there is an error writing the compressed file. 
BZ_OK 

otherwise 


























3.4.7. BZ2 bzWriteClose 
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void BZ2_bzWriteClose( int *bzerror, BZFILEx f, 
int abandon, 
unsigned int* nbytes_in, 
unsigned int* nbytes_out ); 











void BZ2_bzWriteClose64( int *bzerror, BZFILEx f, 
int abandon, 
unsigned int*« nbytes_in_1032, 
unsigned int*« nbytes_in_hi32, 
unsigned int* nbytes_out_1lo32, 
unsigned int* nbytes_out_hi32 ); 


Compresses and flushes to the compressed file all data so far supplied by BZ2_bzWrite. The logical end-of- 
stream markers are also written, so subsequent calls to BZ2_bzWrite are illegal. All memory associated with 
the compressed file b is released. £ flush is called on the compressed file, but it is not fclose’d. 


If BZ2_bzWriteClose is called to clean up after an error, the only action is to release the memory. The library 
records the error codes issued by previous calls, so this situation will be detected automatically. There is no attempt 
to complete the compression operation, nor to £f1ush the compressed file. You can force this behaviour to happen 
even in the case of no error, by passing a nonzero value to abandon. 


If nbytes_inis non-null, *nbytes_in will be set to be the total volume of uncompressed data handled. Similarly, 
nbytes_out will be set to the total volume of compressed data written. For compatibility with older versions of 
the library, BZ2_bzWriteClose only yields the lower 32 bits of these counts. Use BZ2_bzWriteCloseé64 if 
you want the full 64 bit counts. These two functions are otherwise absolutely identical. 


Possible assignments to bzerror: 
BZ_SEQUENCE_ERROR 
if b was opened with BZ2_bzReadOpen 
BZ_TO_ERROR 
if there is an error writing the compressed file 
BAmOK 
otherwise 























3.4.8. Handling embedded compressed data streams 


The high-level library facilitates use of bzip2 data streams which form some part of a surrounding, larger data stream. 


¢ For writing, the library takes an open file handle, writes compressed data to it, {flushes it but does not fclose 
it. The calling application can write its own data before and after the compressed data stream, using that same file 
handle. 


e Reading is more complex, and the facilities are not as general as they could be since generality is hard to reconcile 
with efficiency. BZ2_bzRead reads from the compressed file in blocks of size BZ_MAX_UNUSED bytes, 
and in doing so probably will overshoot the logical end of compressed stream. To recover this data once 
decompression has ended, call BZ2_bzReadGet Unused after the last call of BZ2_lbzRead (the one returning 
BZ_STREAM_END) but before calling BZ2_bzReadClose. 
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This mechanism makes it easy to decompress multiple bzip2 streams placed end-to-end. As the end of one stream, 
when BZ2_bzRead returns BZ_STREAM_END, call BZ2_bzReadGet Unused to collect the unused data (copy it 
into your own buffer somewhere). That data forms the start of the next compressed stream. To start uncompressing 
that next stream, call BZ2_bzReadOpen again, feeding in the unused data via the unused / nUnused parameters. 




















5 


Keep doing this until BZ_STREAM_ 
BZ2_bzReadGet Unused will of course return no data. 








‘ND return coincides with the physical end of file (feof (f£)). In this situation 


This should give some feel for how the high-level interface can be used. If you require extra flexibility, you’ll have to 


bite the bullet and get to grips with the low-level interface. 


3.4.9. Standard file-reading/writing code 


Here’s how you’d write data to a compressed file: 








IIe SEB 

BZFILE* b; 

SLIME nBuf; 

char buf[ /* whatever size you like */ ]; 
sling bzerror; 

int nWritten; 


if == se@jorein (( Vamyarsl ke slows Ua ie 

aac ( fae jy of 

/x handle error x*/ 

} 

b = BZ2_bzWriteOpen( &bzerror, f, 9); 
aie (loxaeicicone Ye IBA Ol) 4 

BAe 7 Wists Cikoisicm( om)r 

/* handle error «/ 


} 


while ( /* condition */ ) { 


/* get data to write into buf, and set nBuf appropriately */ 


igitheeicein = BWA Jove (( wlowsieieoie, lo, lowe, imleieie }) 4 


if (bzerror == BZ_IO_ERROR) { 
BAZ bOzWisitthe Close "(TebZercom, on). 
/x handle error x/ 





BZ2_bzWriteClose( ébzerror, b ); 
if (bzerror == BZ_IO_ERROR) { 
/x handle error «*/ 


} 





And to read from a compressed file: 
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FILEx f; 

BZFILE* b; 

SLIME iQUENOIAE 5 

char buf[ /* whatever size you like */ ]; 
SLIME bzerror; 

int nWritten; 


ie = seojorein ( Yamyicle lows 20, Wiel jp 
ic ( tse } ff 
/x handle error */ 
} 
b = BZ2_bzReadOpen ( &bzerror, f, 0, NULL, 0); 
alee (alo Ah auaxo) a ic )/Amm©).@an mn 
BZ2_bzReadClose ( &bzerror, b ); 
/x handle error */ 


bzerror = BZ_OK; 
while ( bzerror == BZ_OK && /* arbitrary other conditions «/) { 
nBuf = BZ2_bzRead ( &bzerror, b, buf, /* size of buf «/ ); 
if ( bzerror == BZ OK ) { 
/x do something with buf[0 .. nBuf-1] x«/ 





} 
} 
if ( bzerror != BZ STREAM END ) { 
BZ2_bzReadClose ( &bzerror, b ); 
/* handle error «/ 
} else { 
BZ2_bzReadClose ( &bzerror, b ); 











3.5. Utility functions 
3.5.1. BZ2_bzBuffToBuffCompress 


int BZ2_bzBuffToBuffCompress( char dest, 
unsigned int* destLen, 
charx« source, 
unsigned int sourceLen, 
ciate blockSizel00k, 
alae verbosity, 
aioe workFactor ); 
Attempts to compress the data in source[0 .. sourceLen-1] into the destination buffer, dest [0 
xdestLen-1]. If the destination buffer is big enough, *destLen is set to the size of the compressed data, 


and BZ_OK is returned. If the compressed data won’t fit, «dest Len is unchanged, and BZ_OUTBUFF_FULL is 
returned. 


Compression in this manner is a one-shot event, done with a single call to this function. The resulting compressed 


data is a complete bzip2 format data stream. There is no mechanism for making additional calls to provide extra 
input data. If you want that kind of mechanism, use the low-level interface. 
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For the meaning of parameters blockSizel00k, verbosity and workFactor, see BZ2_bzCompressInit. 


To guarantee that the compressed data will fit in its buffer, allocate an output buffer of size 1% larger than the 
uncompressed data, plus six hundred extra bytes. 


BZ2_bzBuffToBuffDecompress will not write data at or beyond dest [*destLen], even in case of buffer 
overflow. 


Possible return values: 
BZ CONFIG _ ERROR 


if the library has been mis-compiled 
BZ_PARAM ERROR 

if dest is NULL or destLen is NULL 

or blockSizel00k < 1 or blockSizel00k > 9 

or verbosity < 0 or verbosity > 4 

or workFactor < 0 or workFactor > 250 
BZ_ MEM ERROR 

if insufficient memory is available 
BZ_OUTBUFF_FULL 

if the size of the compressed data exceeds xdestLen 
BZ_OK 

otherwise 




















3.5.2. BZ2_bzBuffToBuffDecompress 


int BZ2_bzBuffToBuffDecompress( char* dest, 
unsigned int* destLen, 
char* source, 
unsigned int sourceLen, 
SLIME small, 
aerate verbosity ); 
Attempts to decompress the data in source[0 .. sourceLen-1] into the destination buffer, dest [0 
xdestLen-1]. If the destination buffer is big enough, «dest Len is set to the size of the uncompressed data, 


and BZ_OK is returned. If the compressed data won’t fit, «dest Len is unchanged, and BZ_OUTBUFF_FULL is 
returned. 


source is assumed to hold a complete bzip2 format data stream. BZ2_bzBuffToBuffDecompress tries to 
decompress the entirety of the stream into the output buffer. 


For the meaning of parameters small and verbosity, see BZ2_bzDecompressInit. 


Because the compression ratio of the compressed data cannot be known in advance, there is no easy way to guarantee 
that the output buffer will be big enough. You may of course make arrangements in your code to record the size of 
the uncompressed data, but such a mechanism is beyond the scope of this library. 


BZ2_bzBuffToBuffDecompress will not write data at or beyond dest [x*destLen], even in case of buffer 
overflow. 


Possible return values: 
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BZ_CONFIG_ERROR 

if the library has been mis-compiled 
BZ_PARAM ERROR 

if dest is NULL or destLen is NULL 

or small !=0 && small !=1 

or verbosity < 0 or verbosity > 4 
BZ_MEM ERROR 

if insufficient memory is available 
BZ_OUTBUFF_FULL 

if the size of the compressed data exceeds «destLen 
BZ_DATA_ERROR 

if a data integrity error was detected in the compressed data 
BZ_DATA_ERROR_MAGIC 

if the compressed data doesn’t begin with the right magic bytes 
BZ_UNEXPECTED_EOF 

if the compressed data ends unexpectedly 
Bam OK 

otherwise 


















































3.6. z1ib compatibility functions 


Yoshioka Tsuneo has contributed some functions to give better zlib compatibility. These functions are 
BZ2_bzopen, BZ2_bzread, BZ2_bzwrite, BZ2_bzflush, BZ2_bzclose, BZ2_bzerror and 
BZ2_bzlibVersion. These functions are not (yet) officially part of the library. If they break, you get to 
keep all the pieces. Nevertheless, I think they work ok. 

typedef void BZFILE; 











COnsieehicicr Ae 7b 7M Viera Ssie@ mun aZOneGlan) eo 


Returns a string indicating the library version. 
BZFILE * BZ2_bzopen (const char *path, const char «mode ); 
BZFILE * BZ2_bzdopen ( int Cl, const char x*xmode ); 














Opens a .bz2 file for reading or writing, using either its name or a pre-existing file descriptor. Analogous to fopen 


and fdopen. 
shige, [BAe lopaicevevel (( IEWAID ILI lo), \ioakoly loybie, alisne Jkeiay }} 5 
int BZ2_bzwrite ( BZFILE* b, void* buf, int len ); 











Reads/writes data from/to a previously opened BZFILE. Analogous to fread and fwrite. 
alae, IBAA, lo vase lluisie (C ISvaigaibies Joy )) 
AWOL! leva, Joel Lorsiey (( IB valtiLibithes Joy )) p 


r. 




















Flushes/closes a BZF ILE. BZ2_bzflush doesn’t actually do anything. Analogous to fflush and fclose. 
Conse e@hiaie <1 BA wb Zener orw BA bers) setts ne rerum) 





Returns a string describing the more recent error status of b, and also sets «errnum to its numerical value. 
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3./. Using the library in a stdio-free 
environment 


3.7.1. Getting rid of stdio 


In a deeply embedded application, you might want to use just the memory-to-memory functions. You can do this 
conveniently by compiling the library with preprocessor symbol BZ_NO_STDIO defined. Doing this gives you a 
library containing only the following eight functions: 





BZ2_bzCompressInit, BZ2_bzCompress, BZ2_bzCompressEnd BZ2_bzDecompressInit, 
BZ2_bzDecompress, BZ2_bzDecompressEnd BZ2_bzBuffToBuffCompress, BZ2_bzBuffToBuffDecompress 





When compiled like this, all functions will ignore verbosity settings. 


3.7.2. Critical error handling 


libbzip2 contains a number of internal assertion checks which should, needless to say, never be activated. 
Nevertheless, if an assertion should fail, behaviour depends on whether or not the library was compiled with 
BZ_NO_STDIO set. 


For a normal compile, an assertion failure yields the message: 
bzip2/libbzip2: internal error number N. 


This is a bug in bzip2/libbzip2, 1.0.5 of 10 December 2007. Please report it to me at: jseward@bzip.org. If this 
happened when you were using some program which uses libbzip2 as a component, you should also report this bug 
to the author(s) of that program. Please make an effort to report this bug; timely and accurate bug reports eventually 
lead to higher quality software. Thanks. Julian Seward, 10 December 2007. 


where N is some error code number. If N == 1007, it also prints some extra text advising the reader that unreliable 
memory is often associated with internal error 1007. (This is a frequently-observed-phenomenon with versions 
1.0.0/1.0.1). 


exit (3) is then called. 


For a st dio-free library, assertion failures result in a call to a function declared as: 
extern would bzesrmternalbeeraroir (same ermcode™ li 


The relevant code is passed as a parameter. You should supply such a function. 


In either case, once an assertion failure has occurred, any bz_st ream records involved can be regarded as invalid. 
You should not attempt to resume normal operation with them. 


You may, of course, change critical error handling to suit your needs. As I said above, critical errors indicate bugs 
in the library and should not occur. All "normal" error situations are indicated via error return codes from functions, 
and can be recovered from. 


3.8. Making a Windows DLL 


Everything related to Windows has been contributed by Yoshioka Tsuneo (tsuneo@rr.iij4u.or. jp), so you 
should send your queries to him (but perhaps Cc: me, jseward@bzip.org). 
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My vague understanding of what to do is: using Visual C++ 5.0, open the project file Libbz2.dsp, and build. 
That’s all. 


If you can’t open the project file for some reason, make a new one, naming these files: blocksort.c, bzlib.c, 
compress.c,crctable.c, decompress.c, huffman.c, randtable.cand libbz2.def. You will also 
need to name the header files bzlib.h and bzlib_private.h. 


If you don’t use VC++, you may need to define the proprocessor symbol _WIN32. 
Finally, dlltest.c is a sample program using the DLL. It has a project file, dLltest.dsp. 
If you just want a makefile for Visual C, have a look at makefile.msc. 


Be aware that if you compile bzip2 itself on Win32, you must set BZ_UNIX to 0 and BZ_LCCWIN32 to I, in the 
file bzip2.c, before compiling. Otherwise the resulting binary won’t work correctly. 


I haven’t tried any of this stuff myself, but it all looks plausible. 
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These are just some random thoughts of mine. Your mileage may vary. 


4.1. Limitations of the compressed file format 


bzip2-1.0.X, 0.9.5 and 0.9.0 use exactly the same file format as the original version, bzip2-0.1. This 
decision was made in the interests of stability. Creating yet another incompatible compressed file format would create 
further confusion and disruption for users. 


Nevertheless, this is not a painless decision. Development work since the release of bzip2—0.1 in August 1997 has 
shown complexities in the file format which slow down decompression and, in retrospect, are unnecessary. These are: 


¢ The run-length encoder, which is the first of the compression transformations, is entirely irrelevant. The original 
purpose was to protect the sorting algorithm from the very worst case input: a string of repeated symbols. But 
algorithm steps Q6a and Q6b in the original Burrows-Wheeler technical report (SRC-124) show how repeats can 
be handled without difficulty in block sorting. 


The randomisation mechanism doesn’t really need to be there. Udi Manber and Gene Myers published a suffix 
array construction algorithm a few years back, which can be employed to sort any block, no matter how repetitive, 
in O(N log N) time. Subsequent work by Kunihiko Sadakane has produced a derivative O(N (log N)*2) algorithm 
which usually outperforms the Manber-Myers algorithm. 


I could have changed to Sadakane’s algorithm, but I find it to be slower than bz ip2’s existing algorithm for most 
inputs, and the randomisation mechanism protects adequately against bad cases. I didn’t think it was a good 
tradeoff to make. Partly this is due to the fact that I was not flooded with email complaints about bzip2-0.1’s 
performance on repetitive data, so perhaps it isn’t a problem for real inputs. 


Probably the best long-term solution, and the one I have incorporated into 0.9.5 and above, is to use the existing 
sorting algorithm initially, and fall back to a O(N (log N)42) algorithm if the standard algorithm gets into 
difficulties. 


The compressed file format was never designed to be handled by a library, and I have had to jump though some 
hoops to produce an efficient implementation of decompression. It’s a bit hairy. Try passing decompress.c 
through the C preprocessor and you’ll see what I mean. Much of this complexity could have been avoided if the 
compressed size of each block of data was recorded in the data stream. 


e An Adler-32 checksum, rather than a CRC32 checksum, would be faster to compute. 


It would be fair to say that the bzip2 format was frozen before I properly and fully understood the performance 
consequences of doing so. 


Improvements which I was able to incorporate into 0.9.0, despite using the same file format, are: 
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¢ Single array implementation of the inverse BWT. This significantly speeds up decompression, presumably because 
it reduces the number of cache misses. 


e Faster inverse MTF transform for large MTF values. The new implementation is based on the notion of sliding 
blocks of values. 


* bzip2—-0.9.0 now reads and writes files with fread and fwrite; version 0.1 used putc and getc. Duh! 
Well, you live and learn. 


Further ahead, it would be nice to be able to do random access into files. This will require some careful design of 
compressed file formats. 


4.2. Portability issues 


After some consideration, I have decided not to use GNU autoconf to configure 0.9.5 or 1.0. 


autoconf, admirable and wonderful though it is, mainly assists with portability problems between Unix-like 
platforms. But bzip2 doesn’t have much in the way of portability problems on Unix; most of the difficulties 
appear when porting to the Mac, or to Microsoft’s operating systems. autoconf doesn’t help in those cases, and 
brings in a whole load of new complexity. 


Most people should be able to compile the library and program under Unix straight out-of-the-box, so to speak, 
especially if you have a version of GNU C available. 


There are a couple of __inline__ directives in the code. GNU C (gcc) should be able to handle them. If you’re 
not using GNU C, your C compiler shouldn’t see them at all. If your compiler does, for some reason, see them and 
doesn’t like them, just #define __inline_ tobe /* x*/. One easy way to do this is to compile with the flag 
—D__inline__ =, which should be understood by most Unix compilers. 


If you still have difficulties, try compiling with the macro BZ_STRICT_ANSI defined. This should enable you to 
build the library in a strictly ANSI compliant environment. Building the program itself like this is dangerous and 
not supported, since you remove bzip2’s checks against compressing directories, symbolic links, devices, and other 
not-really-a-file entities. This could cause filesystem corruption! 


One other thing: if you create a bzip2 binary for public distribution, please consider linking it statically (gcc 
-static). This avoids all sorts of library-version issues that others may encounter later on. 


If you build bzip2 on Win32, you must set BZ_UNIX to 0 and BZ_LCCWIN32 to 1, in the file bzip2.c, before 
compiling. Otherwise the resulting binary won’t work correctly. 


4.3. Reporting bugs 


I tried pretty hard to make sure bz ip2 is bug free, both by design and by testing. Hopefully you’ll never need to read 
this section for real. 


Nevertheless, if bzip2 dies with a segmentation fault, a bus error or an internal assertion failure, it will ask you to 
email me a bug report. Experience from years of feedback of bzip2 users indicates that almost all these problems can 
be traced to either compiler bugs or hardware problems. 
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* Recompile the program with no optimisation, and see if it works. And/or try a different compiler. I heard all 
sorts of stories about various flavours of GNU C (and other compilers) generating bad code for bzip2, and I’ve 
run across two such examples myself. 


2.7.X versions of GNU C are known to generate bad code from time to time, at high optimisation levels. If you 
get problems, try using the flags -O2 -fomit-—frame-pointer -fno-strength-reduce. You should 
specifically not use -funroll-loops. 








You may notice that the Makefile runs six tests as part of the build process. If the program passes all of these, it’s 
a pretty good (but not 100%) indication that the compiler has done its job correctly. 


If bzip2 crashes randomly, and the crashes are not repeatable, you may have a flaky memory subsystem. bzip2 
really hammers your memory hierarchy, and if it’s a bit marginal, you may get these problems. Ditto if your disk 
or I/O subsystem is slowly failing. Yup, this really does happen. 


Try using a different machine of the same type, and see if you can repeat the problem. 


This isn’t really a bug, but ... If bzip2 tells you your file is corrupted on decompression, and you obtained the file 
via FTP, there is a possibility that you forgot to tell FTP to do a binary mode transfer. That absolutely will cause 
the file to be non-decompressible. You'll have to transfer it again. 


If you’ve incorporated 1ibbzip2 into your own program and are getting problems, please, please, please, check that 
the parameters you are passing in calls to the library, are correct, and in accordance with what the documentation says 
is allowable. I have tried to make the library robust against such problems, but I’m sure I haven’t succeeded. 


Finally, if the above comments don’t help, you’ll have to send me a bug report. Now, it’s just amazing how many 
people will send me a bug report saying something like: 
bzip2 crashed with segmentation fault on my machine 


and absolutely nothing else. Needless to say, a such a report is totally, utterly, completely and comprehensively 100% 
useless; a waste of your time, my time, and net bandwidth. With no details at all, there’s no way I can possibly begin 
to figure out what the problem is. 


The rules of the game are: facts, facts, facts. Don’t omit them because "oh, they won’t be relevant". At the bare 
minimum: 

Machine type. Operating system version. 

Exact version of bzip2 (do bzip2 —V). 

Exact version of the compiler used. 

Flags passed to the compiler. 





However, the most important single thing that will help me is the file that you were trying to compress or decompress 
at the time the problem happened. Without that, my ability to do anything more than speculate about the cause, is 
limited. 


4.4. Did you get the right package? 


bzip2 is a resource hog. It soaks up large amounts of CPU cycles and memory. Also, it gives very large latencies. 
In the worst case, you can feed many megabytes of uncompressed data into the library before getting any compressed 
output, so this probably rules out applications requiring interactive behaviour. 


These aren’t faults of my implementation, I hope, but more an intrinsic property of the Burrows-Wheeler transform 
(unfortunately). Maybe this isn’t what you want. 
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If you want a compressor and/or library which is faster, uses less memory but gets pretty good compression, and has 
minimal latency, consider Jean-loup Gailly’s and Mark Adler’s work, zlib-1.2.1 and gzip-1.2.4. Look for 
them at http://www.zlib.org and http://www.gzip.org respectively. 


For something faster and lighter still, you might try Markus F X J Oberhumer’s LZO real-time compres- 
sion/decompression library, at http://www.oberhumer.com/opensource. 


4.5. Further Reading 


bzip2 is not research work, in the sense that it doesn’t present any new ideas. Rather, it’s an engineering exercise 
based on existing ideas. 


Four documents describe essentially all the ideas behind bzip2: 

Michael Burrows and D. J. Wheeler: 
"A block-sorting lossless data compression algorithm" 
10th May 1994. 
Digital SRC Research Report 124. 
ftp://ftp.digital.com/pub/DEC/SRC/research-reports/SRC-124.ps.gz 
If you have trouble finding it, try searching at the 
New Zealand Digital Library, http://www.nzdl.org. 











Daniel S. Hirschberg and Debra A. LeLewer 

"Efficient Decoding of Prefix Codes" 

Communications of the ACM, April 1990, Vol 33, Number 4. 
You might be able to get an electronic copy of this 

from the ACM Digital Library. 








David J. Wheeler 
Program bred3.c and accompanying document bred3.ps. 
This contains the idea behind the multi-table Huffman coding scheme. 
ftp://ftp.cl.cam.ac.uk/users/djw3/ 


Jon L. Bentley and Robert Sedgewick 
"Fast Algorithms for Sorting and Searching Strings" 
Available from Sedgewick’s web page, 
www.cs.princeton.edu/~rs 


The following paper gives valuable additional insights into the algorithm, but is not immediately the basis of any code 
used in bzip2. 
Peter Fenwick: 
Block Sorting Text Compression 
Proceedings of the 19th Australasian Computer Science Conference, 
Melbourne, Australia. Jan 31 - Feb 2, 1996. 
ftp://ftp.cs.auckland.ac.nz/pub/peter-—f/ACSC96paper.ps 


Kunihiko Sadakane’s sorting algorithm, mentioned above, is available from: 
http://naomi.is.s.u-tokyo.ac.jp/~sada/papers/Sada98b.ps.gz 
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The Manber-Myers suffix array construction algorithm is described in a paper available from: 
http: //www.cs.arizona.edu/people/gene/PAPERS/suffix.ps 





Finally, the following papers document some investigations I made into the performance of sorting and decompression 


algorithms: 
Julian Seward 
On the Performance of BWT Sorting Algorithms 


Proceedings of the IEEE Data Compression Conference 2000 
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Julian Seward 
Space-time Tradeoffs in the Inverse B-W Transform 


Proceedings of the IEEE Data Compression Conference 2001 
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